Skip to content

gh-162056: Keep the initial SRE match context on the C stack - #162057

Open
nityanandmathur wants to merge 6 commits into
python:mainfrom
nityanandmathur:sre-initial-context
Open

nityanandmathur wants to merge 6 commits into
python:mainfrom
nityanandmathur:sre-initial-context

Conversation

@nityanandmathur

@nityanandmathur nityanandmathur commented Nov 15, 2026 •

Copy link
Copy Markdown

Related issue: #162056.

Change

Keep the first SRE(match_context) local to each SRE(match) invocation. Offset -1 identifies this context; nested contexts and saved marks continue to use the existing resizable data stack. Return from a nested/recursive-count path restores either the local context or the offset-addressed nested context. Initial-context exit does not pop stack storage that it did not allocate.

The implementation change is intentionally limited to Modules/_sre/sre_lib.h, with one regression test in Lib/test/test_re.py and a Library NEWS entry. No pattern syntax, opcode or public API change is intended. The regression exercises ignorecase-set count fallback, nested repeats, greedy/lazy/possessive repeats, bytes/Unicode and scanner iteration.

Review focus: one macOS findall benchmark showed an unresolved ~1.6% slowdown. Feedback on context lookup and C-stack usage is welcome.

Measurements (historical builds, not current-head performance)

All historical timing below uses CPython ce841a9a02b9a28367b52b7b0c9606b5bdeacada. Ratios are baseline time / candidate time; values greater than 1 favor the candidate.

Platform/build Pygments highlighting Other findings
Linux x86-64, GCC 14.2, two independently trained full PGO builds (--enable-optimizations --without-static-libpython; no LTO requested) 1.243× median for a five-file highlighting/HTML operation, descriptive 95% interval 1.163–1.317×; 11/12 groups favor candidate Same-binary A/A median 1.017×, with nonzero bias; packaging inconclusive
Native macOS, original matched build pair 1.231–1.246× across five individual source inputs; each favors candidate in 12/12 groups findall 0.983969×: about 1.63% more elapsed time; 9/12 groups slower. A/A 1.001005×. The warning is not removed by a post-hoc multiple-comparison analysis
Windows x64, MSVC 19.44.35230 / SDK 10.0.26100, matched PGO (/O2 /GL /LTCG /USEPROFILE; 43-test training, seed 724313) 1.2162–1.2700× across individual cases findall 0.9787× and substitution 0.9558×; both descriptive intervals cross 1. Three SRE A/A quality flags remain; one Pygments interval also crosses 1

An earlier Windows non-PGO Release run gave Pygments point estimates 1.179–1.232×, but had short CPU sampling windows and noisy controls; it is preliminary and not pooled with PGO. PGO removes the short/zero-CPU-window flags, not all A/A variation. The Windows harness is a custom worker, not pyperf.

These are one compiled pair per experiment, not independently rebuilt replications for every timing group. Inputs differ across platforms. Repeated inner samples are not independent experiments, no observations are discarded, and these results do not establish an overall Python/regex or whole-Sphinx improvement. Raw historical reports are retained privately and are not presently public reproducibility artifacts.

Correctness

Historical coverage includes:

  • Linux PGO: 2,447 filtered tests across test_re, test_textwrap, test_argparse, test_json, plus deterministic differential corpora. Separate historical debug/ASan/UBSan pairs passed the same selection; LeakSanitizer was unavailable and no leak-clean claim is made.
  • Native Mac: both binaries passed 2,448 focused tests and matching 2,814-record semantic checks.
  • Windows: focused suites and semantic checks passed for the matched builds. These are not full CPython suite results.

Fresh validation at base 369d8bc4f6fe1f751de2512523a1acb95491a5eb, with this branch's exact patch:

./configure --with-pydebug
make -j6
./python -m test test_re test_regrtest -j2
./python -m unittest test.test_re.ReTests.test_repeated_ignorecase_set_context

Linux x86-64/GCC 14.2 baseline and candidate builds succeed. The combined runs execute 316 and 317 tests respectively, with four skips each. test_regrtest passes on both. Both runs fail only at test_regression_gh94675 because this environment forbids its multiprocessing AF_UNIX socket (EPERM). No test was changed or excluded in these fresh runs. The candidate's added regression passes independently. This is not an all-green unfiltered result or current-head performance confirmation.

Remaining work

Independent release-like runs on current main; public reproducibility artifacts; investigation of the Mac findall result; broader mixed-regex/application workloads; free-threaded and 32-bit coverage; fixed C-frame cost and error-path review. Unbounded backtracking still uses the explicit heap stack, but each actual C invocation pays the larger local frame.

AI tools assisted the implementation, test execution and analysis.

@python-cla-bot

python-cla-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@bedevere-app

bedevere-app Bot commented Nov 15, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@nityanandmathur
nityanandmathur marked this pull request as ready for review November 15, 2026 17:32
No source changes. Allow the normal upstream CI checks to validate this
proposal after the initial commits intentionally skipped CI.
@ZeroIntensity ZeroIntensity added the AI-needs-explanation AI-assisted work that needs further explanation and follow up from the contributor label Nov 15, 2026
@bedevere-app

bedevere-app Bot commented Nov 15, 2026

Copy link
Copy Markdown

Thank you for your interest in contributing to Python. This is a community-driven project, and we want it to be a welcoming place to learn and collaborate.

Contributors are responsible for reviewing and understanding what they submit, checking its claims, and being able to explain changes in their own words. This helps reviewers focus on the contribution and leaves more time to support other contributors.

This contribution appears to have been authored with AI assistance. Please explain your contribution in your own words and describe how you checked its correctness.

Please read our guidelines for using AI tools.

@nityanandmathur

Copy link
Copy Markdown
Author

I reviewed the patch and used AI assistance during implementation and testing.

The change is in SRE(match) in Modules/_sre/sre_lib.h. Previously, its initial SRE(match_context) was reserved on state->data_stack via DATA_ALLOC. The patch instead creates an automatic initial_ctx and points ctx to it, with ctx_pos == -1 identifying that context.

Nested contexts remain on the resizable data stack and use non-negative offsets. DO_JUMPX saves the parent’s offset in last_ctx_pos; DATA_STACK_LOOKUP_AT now resolves -1 to initial_ctx and other offsets relative to state->data_stack. This matters because growing the data stack can relocate its backing buffer, invalidating pointers into it.

On unwind, the initial context returns directly after updating state->sigcount; nested contexts still pop their stack allocation, restore their parent and dispatch through the saved jump value. The patch doesn’t replace the existing backtracking stack with C recursion.

One path I paid particular attention to is the recursive fallback from SRE(count): it can call SRE(match) and grow the shared data stack, so the caller must restore the correct context after returning. The added regression test exercises case-insensitive sets with greedy, non-greedy and possessive repeats, nested repeats, capture values and spans, and repeated scanner searches through exhaustion. It covers both str and bytes, including strings with different internal character widths.

I had correctness and performance tests run with AI assistance. The PR’s CI jobs have passed. The speedup is workload-dependent: highlighting benchmarks improved, but a small macOS findall regression remains under investigation and is disclosed in the PR. I’m not claiming a universal speedup.

@nityanandmathur

Copy link
Copy Markdown
Author

@NekoAsakura Could you please review this or assign the relevant person for a quick review?

@NekoAsakura

Copy link
Copy Markdown
Member

The PR queue is long and there aren't many reviewers, so please wait as the Devguide explains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-needs-explanation AI-assisted work that needs further explanation and follow up from the contributor awaiting review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants